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ABSTRACT 



What is disclosed is method that interpolates missing color 
components for an original pixel by utilizing color medians 
for surrounding blocks of green pi™»i«= a n»w pj^j jc miipit 
which has complete col or informa tion which is a function o f 
thftnrifnnai pivfii anrj inlflrfl plMed color componen ts. The 
interpolated pixel is a true representation of both luminance 
and chrominance information. 
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EFFICIENT ALGORITHM FOR COLOR 
RECOVERY FROM 8-BIT TO 24-BIT COLOR 
PIXELS 

This application is a divisional of U.S. patent application 
Ser. No. 08/963,525, filed Nov. 3, 1997, now U.S. Pat. No. 
6,091,851, Jul. 18, 2000. 

BACKGROUND OF THE INVENTION 

In digital cameras and imaging devices using only a single 
electronic sensor, the image color is captured by sub- 
sampling typically in three (Red, Green, Blue) color planes. 
In such cases, the original "raw image" is captured with each 
"raw image pixel", composed of only one of three vital 
color/intensity components: either R, G, or B (Red, Green, 
or Blue). But for image rendering devices (such as a monitor 
or printer), full color resolution is desirable. Recovery of full 
color information from this sub -sampled color is popularly 
known as color interpolation. 

The following description refers to raw image pixels as 
either "pixels" or "raw image pixels". Pixels that are formed 
b y virtue of a completed color interpolation techn ique wjll 
he referred tn as "full color pixels" . In digital still and video 
cameras and certain other imaging devices, raw f mages are 
jirst represented as rectangular row and colum n nf pixels 
with each pixel having the intensity value of a particula r 
color nr^ y, In the case of RGB (Red, Green and Blue) 
sub -sampling imaging devices, images are obtained and 
stored in a "Baver" pattern. T he bayer pattern, when three 
color planes are sub-sampled using a color filter array 
(CFA), is characterized by including on a first row of an 
image, alternating pixels of Red and Green, and then on a 
next row alternating pixels of Blue and Green. Thus, the 
Bayer pattern for the first four rows of pixels (with each 
pixel typically an 8-bit value) is as follows (with the rows 
thereafter repeating the same pattern): 

G R G R G R ... 

B G B G B G ... 

G R G R G R ... 

B G B G B G ... 



As a result, each pixel location contains a value for a 
single color only. Assuming, as is the case in some imaging 
devices, that each pixel of the Bayer pattern has 8 bits of 
resolution (i.e., the pixel is a value representing the intensity 
of the color ranging from 0-255), then a "full color" pixe l. 

_pne having R Q unrl B ™™p™™t c *xm,i1^ fa q ?4-h|f y fl |„ ft 

To recover the two missing color components for each pixel, 
a color interpolation or color recovery is employed. 

_In color interpolation, an attempt is made to combine i n 
some fast] ton neighboring pixel(s) to reconstruct , recover, or 
predict missing color component information for a given 
raw image pixel. Several prior art interpolation methods are 
described in "Interactions Between Color Plane Interpola- 
tion and Other Image Processing Functions in Electronic 
Photography", James E. Adams, Jr., SPIE, (The Interna- 
tional Society for Optical Engineering), Vol. 2416 
(Electronic Imaging '95), pages 144-155 (July 1995), with 

rnort nf thftCP. pmplnyinp nive.l av^rapinp; techniques to 

recover missing color components . At its simplest, a prior art 
averaging technique would interpolate the Red component 
of the captured Green pixel at row 2, column 2, (referring to 
the 8-bit Bayer pattern above) as the average of the two 
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nearest red pixels, i.e., Red pixel at row 1, column 2 and the 
Red pixel at row 3, column 2. Likewise, the Blue component 
of the same captured Green at row 2, column 2 would be 
interpolated under an averaging technique as the average of 

5 the two nearest Blue pixels, i.e., the pixel at row 2, column 
1 and the Blue pixel at row 2, column 3. The Blue and Red 
interpolated components are then combined with the original 
captured Green value of the pixel, row 2, column 2 to form 
a new pixel which is composite "full" color pixel. The 

to resultant "RGB" full color pixel is an approximation or 
prediction of the original object's true color at that relative 
pixel location. 

This averaging technique is crude in that the most visually 
critical information, such as edges, are decimated. If the 

is pixel whose missing components are being interpolated lies 
on a edge boundary, then the averaging of pixels of corre- 
sponding color on each side of the boundary may cause that 
component to be a highly inaccurate version of the actual 
uncaptured color component. The edge, for that component 

2° would thus disappear. The response of the human visual 
system (eye, brain, etc.) to edge information is very acute, 
and thus, the blurriness of edge(s) will greatly detract from 
the perceived overall quality (sharpness, resolution, etc.) of 
an image. Further, other averaging techniques that attempt to 

25 detect patterns or edges are complex to implement in high- 
speed rendering applications. In these techniques, different 
formulas for averaging known as predictors are employed 
even within the same image depending on detected local 
patterns within the image. These techniques are not very 

30 efficient and further, if color components are interpolated 
when the pixels are being output, these techniques slow 
down the process of displaying or rendering pixels of an 
image into a display device. Averaging techniques produces 
images that are "soft" and not very crisp when viewed. 

3 5 Further "luminance" information contained in the Green 
pixels (which comprise 50% of the image according to the 
Bayer pattern) are a less significant factor than they should 
be when interpolation is carried out in the prior art tech- 
niques. The luminance information is greatest in Green 

4 0 pixels since Green is already a combination of two primary 
colors, Yellow and Blue. Thus, in terms of visual perception, 
Green contains more information about the original color of 
an area of pixels or pixel in the image than do Blue or Red 
pixels. 

45 Thus, there is a need for an efficient color interpolation 
technique that more accurately reconstructs the original 
object color by taking into better account luminance infor- 
mation contained in Green pixels. 

50 SUMMARY OF THE INVENTION 

An interpolation method is disclosed for a raw image 
composed of pixels that utilizes color medians of surround- 
ing blocks of Green pixels in order to select neighboring 
pixels to attain missing color components for a raw image 
55 pixel. A new pixel is formed that has full color information 
which is a function of the raw image pixel and the interpo- 
lated color components. 

BRIEF DESCRIPTION OF THE DRAWINGS 

60 The objects, features and advantages of the method and 
apparatus for the present invention will be apparent from the 
following description in which: 

FIG. 1 is a flow diagram of color interpolation method- 
ology according to one embodiment of the invention. 

65 FIG. 2 is an exemplary pixel map illustrated to explain 
color interpolation according to one embodiment of the 
invention. 
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FIG. 3 is a system diagram of one embodiment of the statistical median X, the set [X 1? . . . , X v ] must be an ordered 

invention. set of values. If each color block is chosen to contain only 

FIG. 4 illustrates a simplified methodology for determin- Green raw image pixels, then regardless of the color (R, G, 

ing color median according to one embodiment of the or B) of the component being interpolated, only Green raw 

invention. 5 i ma 8 e pixel values are considered in calculating the color 

median. This more significantly utilizes luminance informa- 

DETAILED DESCRIPTION OF THE tion in making the selection of which raw image pixel is to 

INVENTION be used as the missing color component. In essence, the 

n r ■ . *i_ * i ~u j ™ i c *u median for one block of Green pixels, of which there are 

Rcfcrnng to the figures, exemplary embedments of the four ^ ^ • of 

invention will now be described. The exemplary embodi- 10 r t t - . j • r *u oMn « • 

•j j t .« * r lL * j [except at image boundaries J in the 8-bit Bayer pattern, is 

ments are provided to illustrate aspects of the invention and L v A . , . * , *u wi i * r- 

, t , f4 . • compared against color medians for other blocks of Green 

should not be construed as limiting the scope of the inven- r . ■ i • j 4 •• i_* L _r 

™ . ... t • -I j *uj raw image pixels in determining which surrounding raw 

tion. The exemplary embodiments are primarily described r 1 r . i iL • • i 

... r f ui i j- n u * a * 4L image pixel to select in interpolating the missing color 

with reference to block diagrams or flowcharts. As to the * xuu n • ■ 1 • * * 

flowcharts, each block withto the flowcharts represents both 15 ^P^f • ™»» * ^ components are inter- 

. , ' , *. i . r c ,l polated for each raw image pixel, these components are 

a method step and an apparatus element for performing the r , . 6 . , f t „ , * , 

. j , *L -m • . , , 4 . 6 combmed with the raw image pixel to form a full color pixel 

method step. Depending upon the implementation, the cor- ^ ^ as ^ (j and g information 

responding apparatus element may be configured in ' 



hardware, software, firmware or combinations thereof. 



One embodiment of the invention is in FIG. 1. In this 

, . e i • i * . ^au embodiment, a Green raw image pixel is treated slightly 

FIG. 1 is a flow diagram of color interpolation method- ^ n n , , nl ° r . , . 

& . /- , • differently than Red and Blue raw image pixels when 

oloey according to one embodiment or the invention. . . i *• *u • • i * t: *u* 

&J & mterpolaUrig its other missing color components. For this 

The improved method of color interpolation disclosed reason, the methodology first checks if the raw image pixel 

herein is advantageous in better overcoming the blurriness p for whom missing components are being interpolated 

effect over averaging-type interpolation techniques and also, ^ ^ Green or not Green ( step 110 ) Since a Green raw 

in relating each raw image pixel's missing components to pixd already con tains its own luminance information, it is 

more relevant color information. Thus, various embodi- treated differently than a non-Green raw image pixel such as 

ments of the invention define a parameter referred to as the Red which must obtain i um i nance information from sur- 

"color median", and based on the color median, the tech- rounding pixels 

nique selects one neighboring pixel (with the same color as 3Q Jf ^ ixel ig nQt Green> ^ a Green onent ^ well 

the sought-after missing color component of the raw image as Qne Qf ^ Red 0f BJue colof onems is missing in 

pixel under consideration) over another same color neigh- ^ - { ^ missi Green onent will be interpo i ate d 

boring pixel to be interpolated as the missing color compo- as ^ medjan of adjacent GreeD ixels ( 12Q) Given an 

nent. The utilization of the color median allows the intelli- g _ bU fi ^ { ^ ^ {h& Gfeen onent 

gent selecuon of an entire neighboring pixel value to 35 ^ be mte lated as a median of north> souihj east and 

compose the missing color component. In the averaging- west adjacent ^ yahlcs (sce pj G 2 mapping) whel her the 

type techniques of interpolation, an averaging of values that ^ p for which color mmp0Mnts are beillg interpolated is 

are highly uncorrected from one another creates pixel a Bluc Qr Red ixd ^ faw ^ M p for whkh 

artifacts by introducing colors not guaranteed present in the missi mmpoaesas are being interpolated, has only one 

raw image By contrast, vanous embodiments of the inven- 4Q final ^ siacc Grcen is interpolated at step 

tion describe selecting one of the neighboring pixels as the UQ ^ since ^ M p aJread CQntains Qne of either Red 

missing value, based on image characteristics such as color 0f filue cobr m f ormationi 

median (defined below). This improves the overall visual „ . t , 4 . . n A */•<• ■ i n • 

... v r iL / 1 _ „ r , , To interpolate the missing Red component (il pixel P is 

quality of the image, better preserves edges and more _ m ^ m B , ue , (tf ^ p fc Red) , he 

accurately represents colors present the original object „„ ' . t , . °. r • * i j- i 

.. . J r , . . r . , . ? .-t . 45 first step is to determine the appropriate color median values, 

which was captured by the imaging device. Artifacts are „ r . . r , f ^ 

j. ..... 4 . i4j ^ & ,. ,„ , Thus, according to step 130, the color median for the Green 

diminished since the selected component is actually present , , / j. . r . n , t r 

. 4 . , . . , . r J r block surrounding pixel P and the color median tor the 

in the captured original image Green blocks surrounding each adjacent non-Green pixel is 

The techniques discussed below in the vanous embodi- dctcrraincd In a Baycr pat tern, a non-Green pixel P will 

ments of the invention determine the color median for each 5Q faave 4 & n adjacent ixels of me same color as the 

block of pixels surrounding the pixel under consideraUon for missifl oncQi being i n t erpo iated. Thus, if pixel P is 

selecuon. Essentially, the pixel surroundmg the block of Red (i missi a Blue co onent)) the northw e Stj 

pixels with the "closest" color median to the color median of south ^ est> northeast and southeast adjacent pixels will by 

the block of pixels surrounding the raw image pixel for defauk be Bhie ixeJs (&ee pj G 2 ). The color median for a 

which the missing component is being interpolated is chosen S5 Wock Qf Green m ^ embodiment of me inven tion, 

to be that missmg component. ^ determioed for each of those adjacent non-Green pixels 

In one embodiment of the invention, an ordered set of four ( step ^ 0 y 0nce all color med i ans have been determined, 

pixels {X,, Xa, X 3 , X4}, particularly wherein each X 1( ^ thc ncxt step ^ to fiad tac pixcl whose surround i ng b l oc k's 

X 3 , X 4 is a Green pixel, constitutes a "block" for whom color median is the "closest" to the color median of the block 

color median is determined. The pixel block {X 1( X^, X3, 6Q Slirroun( iing p ix e i p ( step 140). The term "closest" as used 

XJ surrounds adjacendy the raw image pixel for which herein when describing the various embodiments of the 

color median is determined. The "color median" for a block mven tion refers to one value that has the minimum absolute 

of pixels is represented hereinafter as X. When X 1( X^ . . . , value distance (difference) to another value. Thus, each color 

Xjv is an ordered set of values such that each X. ^X,^ for all median for blocks surrounding the adjacent non-Green pix- 

i in the range [1, N-l], the median X of the set [X x . . . X N ] 65 els is subtracted from the color median of the block sur- 

is M such that if N is odd M=X ( ^_ J)/2 and if N is even rounding pixel P. The minimum of these differentials, in an 

M=(X JV/2 +X M2+1 )/2. To guarantee these values M are the absolute value sense, is the "closest" color median to that of 
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the block surrounding pixel P. The final missing component Interpolation that utilizes the color median of Green pixel 
is interpolated to be the adjacent raw image pixel whose blocks, as in this and other embodiments of the invention, 
surrounding block color median is closest to the color better represents luminance in recovering missing color 
median of the block surrounding raw image pixel P (step components. The resulting R, G, and B components when 
150). Finally, with each R, G, and B accounted for, the 5 combined to form each full color pixel will visually more 
value(s) for each color component is/are stored to a storage closely resemble the original color of the object prior to its 
medium or output to be rendered to a display device to form image capture into the Bayer pattern CFA 
a full color pixel (step 160). The process of interpolation may be performed when the 
Referring again to step 110, if the pixel P is Green, then image is about to be displayed onto a monitor or output to 
a different methodology is used to interpolate the missing :0 a peripheral device such as a printer. The methodology 
Red and Blue components. Since both Red and Blue com- shown in FIG. 1 may be implemented in computer software 
ponents are missing, they may be interpolated in any order, such that an 8-bit Bayer pattern CFA image, when its pattern 
but the methodology shown in FIG. 1 interpolates the Blue values are read out, can also be interpolated in a pixel-by- 
component first. According to step 125, first, the color pixel fashion. Often, the 8-bit Bayer pattern CFA data will 
median of blocks surrounding adjacent Blue pixels is deter- have been stored in a compressed format, such that the data 
mined. In a Bayer pattern as shown in FIG. 2, a pixel P that 15 miist fi^t fc e decompressed to recover the Bayer pattern 
is Green will ordinarily have north and south neighboring values. After interpolation, the 24 bit-per-pixel resulting 
adjacent pixels of Blue color. Thus, two color median RGB reS olution is convenient for most display envi- 
values, one for the Green block surrounding the north ronmeil ts currently commercially available. However, the 
adjacent Blue pucel and one for the block surrounding the methodology of Fia x can be adapted t0 any raw image 
south adjacent Blue pixel, are determined. Next, the Blue 20 ^ a ^ m or 

component is interpolated to be the value of the adjacent . . j j . 7. ■ j ■ j ■ i i i 

tii * i *u <l i j • , 4U i i ■ i can be extended to attain any desired pixel color resolution 

Blue pixel with the color median closest to the value of pixel , ol / . . ~ . 

P (step 135). This is in contrast to the methodology a u nd " u Qot ***** * ™ S ' b « t0 24 " blt ^version. Further, 

employed when pixel P is not Green in that the color median lhou 8 h om V ^mediately adjacent pixels have been used to 

of the non-Green pixel P is used rather than the value of 25 intelligently select the interpolated value of a missing color 

pixel P itself when being compared to neighboring pixel component, one skilled in the art will readily be able to 

color medians. When pixel P is Green, the pixel itself is the modify the methodology of FIG. 1 to expand the selection 

best representation of luminance of the pixel for which set to pixels not immediately adjacent but two or more pixel 

missing components can be selected and thus, is used to locations away in distance from the pixel location whose 

compare against neighboring block color medians. 30 missing color components are being interpolated. 

Next, the color median of blocks surrounding adjacent FIG. 2 is an exemplary pixel map illustrated to explain 

Red pixels is determined (step 145). As described above color interpolation according to one embodiment of the 

with respect to adjacent Blue pixels, in a [8-bit] Bayer invention. 

pattern such as that illustrated in FIG. 2, there are two such FIG. 2 shows a 15x15 Bayer pattern of a typical image 

adjacent Red pixels for whom Green block color medians 35 with 8 -bit color information in each pixel location generated 

are determined. Next, according to step 155, the distance, in in a Color Filter Array (CFA) from an image sensor. The 

an absolute value sense, between each color median for methodology described above with respect to FIG. 1 will be 

blocks surrounding the Red pixels and the pixel P is deter- illustratively explained with regard to the raw image pixels 

mined. The Red component of pixel P is interpolated to be shown in FIG. 2. The terminology X(m,n) will indicate a 

the value of the pixel whose surrounding block color median 40 pixel X at row location m and column location n. The pixel 

is closest in distance to the value of pixel P (step 155). Once will be represented by either an R, G, or B referring to its 

all R, G, B values are accounted for a given pixel P, then the original color from the CFA. Neighboring pixels will be 

R. G, B values for the pixel are stored or output (step 160) designated by the pixel color and a subscript indicating 

to form a full color pixel. This interpolation technique forms direction from the pixel X. Where no directional indicator is 

a full color RGB pixel, and assumes the same resolution 45 appropriate, pixels will be represented by color and sequen- 

(such as 8 bits) for the raw image pixel and each of the tial numerical identifier, e.g., Gl, G2, etc. 

interpolated components. For example, consider the Blue pixel in location (6,3) i.e., 

Processing of steps 110-160 will repeat for every pixel in row 6 and column 3. The pixel B(6,3) consists of Blue 
until all pixels in the raw image have been assigned R, G, color information only. It is thus desirable to recover the 
and B values. In the case of an 8-bit Bayer pattern CFA, each 50 missing Green and Red color component information for 
newly formed full color pixel may have a total resolution of B(6,3) such that the pixel location (6,3) can become a full 
24 bits. Thus, according to step 170, this methodology color (RGB) pixel. The pixel B(6^) has four neighboring 
checks if any more raw image pixels are remaining that have green pixels Gn (the north neighbor), Gs (south), Ge (east), 
missing color components to be interpolated. If there are no Gw (west) and four neighboring red pixels Rne (the north- 
more pixels that have missing color components (checked at 55 east neighbor), Rse (southeast), Rsw (southwest) and Rnw 
step 170), then the interpolation process will have been (northwest), as shown in FIG. 2. The neighboring Green 
completed for a given image. pixels form the green color block represented by the vector 

The newly formed full color (RGB) pixels may be stored G={Gn, Ge, Gs, Gw} whose color median is G, the color 

in an array or output (step 160) one by one to be rendered median of the block surrounding B(6,3). The missing Red 

onto a display device. The full color pixels may be output for 60 component R for pixel location (6,3) will be interpolated by 

rendering on-the-fly as soon as each pixel is interpolated selecting one value from the set of whole surrounding pixel 

rather than waiting for all pixels to attain full color. Until the values Rne, Rse, Rsw and Rnw. The selection is based upon 

interpolation process is complete for all pixels, the original the values of color medians of the four neighboring green 

8-bit Bayer pattern, at least in the relevant part, should be color blocks Gne«{Gl, G2, Ge, Gn}, Gse={Ge, G3, G4, 

maintained and not overwritten in the medium where stored 65 Gs}, Gsw-{Gw, Gs, G5, G6}and Gnw-{G8, Gn, Gw, G7} 

since the interpolation procedure may use a raw image pixel centering around pixels Rne, Rse, Rsw, Rnw, respectively, 

more than once before all pixels attain full value. The technique selects the pixel whose block color median is 
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closest to G. The Red color pixel contained at the center of determine An-G-Gn, As-G-Gs, Aw-G-Gw, Ae-G- 

the green color block with closest color median to G will be Ge 

interpolated as the Red component for the pixel location If ( An|<|As[) then P(Blue)oBn else P(Blue)«Bs; 

(6,3). The missing Green color component will be interpo- If ( Aw|<|Ae|) then P(Red)oR n else P(Red)=Rs; 

lated as the median of the four green pixels {Gn, Ge, Gs, 5 } 

Gw} surrounding pixel location (6,3). Thus, when these end 

missing components are determined, the pixel location (6,3) 1° toe above pseudo code, x in P(x) represents the sought 

will have Green and Red color component information after rnissing color component. When the above pseudo code 

which can be combined with the Blue value of the raw image * In tbe above P seudo x m p 00 ^presents the sought 

pixel at (6,3) to form a new full color pixel. 10 after missing color component. When the above pseudo code 

y . « j ■ j , At _ is implemented as computer program product, several cor- 

Li Ice wise, the same procedure can be used to recover the n r A - » . j 4 . . ' 

. . ! ir -ii ollary steps for more efficiently determining the median as 

missing Blue color component for a pixel location contain- , ' . V™ A , / , 

■ • ii ti j i • r i r- i iL shown in FIG. 4 may be employed, 

ing originally Red color information only. For example, the j ma e Boundaries 

pixel at location (3,10) contains Red information only and ^ J* r . . , ™ A r ~ T ~ - . , , 

f, i , . . , n . . ,t ■ 15 Referring again to the CFA of FIG. 2, pixel locations at 

thus needs to attain Green and Blue components through , j • <?.u u .u • ^ ji C 

, A . ™ ni , ? * nr<% in\ • toe boundaries of the image, such as those in rows I and 15 

interpolation. The Blue color component of R(3,10) is j . . j *<r j 4 L i * * cc 

, r - rT , « „ jn ui- • and columns 1 and 15 do not have a complete set of four 

chosen from one of Bne, Bse, Bsw and Bnw neighboring . . . . , . . ^ . . , . , . . . . 

, , ,. \t_ % c iL * » • neighboring adiacent Green pixels as do the pixels at interior 

values, depending upon the values ot the color medians . ° . % c " i *■ -»\ tm_ i 

r to r - locations of the array such as location (6,3). The color 

Gne-median{Gl, G2, G3, Gn}, Gse-median{Ge, G3, G4, 20 inte rpolation methodology can include simplified color 

G2}, Gsw-median{Gw, Gs, G5, G6} and Gnw-median{G8, median techniques for certain image boundary pixels. Thus, 

Gn, Gw, G2} centering around the Blue pixels Bne, Bse, a t a corner pixel location (1,1), the Red component can be 

Bsw, and Bnw respectively. interpolated as the only adjacent Red pixel which is at 

Recovery of the missing Blue and Red color components location (1,2). Likewise, the Blue component from the 

for a raw image pixel location containing Green (G) is 25 corner pixel (1,1) can be interpolated as the only adjacent 

slightly different from that of a non-Green pixel. The raw Blue pixel which is at (2,1). 

image pixel at location (11,9) contains Green originally and For other edge pixels such as the Red pixel at location 

thus it would be desirable to interpolate missing Blue and (1,2), the Green component is the median of the three 

Red components. To recover the Blue color component, the adjacent Green pixels rather than a block of 4 pixels as with 

color medians Gn (for pixel Bn) and Gs (for pixel Bs) are 30 interior pixel locations. Determining the median of three 

Gn=median{Gl, Gne, G(11,0), Gnw} and pixel values involves less computation than does calculating 

^ j • r^>/^ o\ ^ ^ i i the median of four pixel values. The median of pixels |X 1( 

Gs=median{G(ll,9), Gse, G3, Gsw}, respectively, are ^ x 3 } ^ simply ^ provided that me set {Xi) ^ x ^ } ^ 

determined. If the absolute value difference between Gn and ordered from lowest to highest> resp ectively. 

G(ll,9) is smaller than the absolute value difference 35 The missing Blue component of the Red pixel at location 

between Gs and G(ll,9), Bn is selected to be the Blue color (1,2) must be selected between the Blue pixel at (2,1) and 

component at location (11,9), otherwise Bs is selected to be B(2,3), the Blue pixel at location (2,3). There is a block of 

the Blue color component at that location. three surrounding Green pixels used to determine color 

To recover the missing Red color component, the tech- median for pixel B(2,l) and a block of four Green pixels for 

nique determines the color medians Gw (for pixel Rw) and 40 toe block surrounding pixel (2,3). Thus, the color median for 

Ge (for pixel Re) which are Gw-median{Gnw, G(ll,9), thc pixel B(2,l) will be the middle value when the set of 

n n A \ Ah a- m n n\ V / P^ 1 values G ( 1 » 1 )> G ( 2 > 2 ) and G ( 3 ^) is ordered from 

Gsw, G4} and Ge-median{Gne, G2, Gse, G}, respectively. f . . , \ Al u u u a • re #• 

' J ~ J lowest to highest. Other such boundary simplifications can 

If the absolute value difference between Gw and G(ll, 9) is be employed by one of ordinary skill in the art as is 

smaller than the absolute value difference between Ge and 45 required/desired by the arrangement of periphery pixels. 

G(ll,9), Rw is chosen to be the red color component for FIG. 3 is a system diagram of one embodiment of the 

location (11,9), otherwise Re is chosen. invention. 

In pseudo -code, the entire methodology can be described FIG. 3 illustrates a computer system 310, which may be 
as follows: any general or special purpose determining or data process- 
begin 50 ing machine such as a PC (personal computer), coupled to 
if the pixel location P is not Green then a camera 330. Camera 330 may be a digital camera, digital 
r video camera, or any image capture device or imaging 

(Green)=median{Gn, Ge, Gs, Gw}; svstem > and is utnized t0 ca P ture a sensor ima S e of an ob i ect 
, . . . f, a f, a f. ^ xa n 340. Essentially, captured images are compressed and pre- 
determine Al=P-Gne, A2«P-Gse, A3«P-Gsw, A4»P- „ , . . J * . 6 . frU t tU y 

'55 cessed by an mi age processing circuit 332 so that they can 

. G QW ; be efficiently stored in an image memory unit 334, which 

Find A m ^-min{|Al|, |A2|, |A3|, |A4|}; may be a or other storage device such ^ a fixed disk> 

If (A m ^,HAl|) then P(Red>Rne if Red is missing, else miniature card, etc. In most digital cameras, raw images 

P(Blue)-Bne if Blue is missing; obtained from the image capture mechanism such as a 

If i^T^ ^1 VQte*)?*?* * Red 15 missin g> 60 sensor are stored first on camera 330 and downloaded later 

P(Bhie)-Bse if Blue is missing; to be output> displayed) or manipulated. This allows the 

If (A mto -|A3|) then P(Red)-Rsw if Red is missing, camera 330 t0 capture ^ next object quickly without 

P(Blue)*Bsw if Blue is missing additional delay. One convenient raw image representation 

If (A m(M =|A4|) then P(Red)=Rnw if Red is missing, fe the 8 _ bit Bayer patterQ CFAshown in p ia 2 . 

P(Blue)=Bnw if Blue is missing; 6S In one embodiment, an image is captured by camera 330 

}; as a raw 8-bit sub -sampled Bayer pattern CFA. Color 

if the pixel location P is Green (G) then interpolation in this embodiment of the invention operates as 
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follows. First, the sub -sampled image is downloaded from 
camera 330 through I/O port 317 to a disk 318 or other 
device on the computer system 310. The computer system 
performs color interpolation by executing instructions for 
doing so according to various embodiments of the invention, 5 

The methodology described in the various embodiments 
of the invention may be executed using a processor 312 such 
as the PentiumS (a product of Intel Corporation) and a 
memory 311, such as RAM, which is used to store/load 
instruction, addresses and result data. The application used 10 
to interpolate the color information of the image may be an 
executable module compiled from source written in a lan- 
guage such as C++. The instructions of that executable 
module, which correspond with instructions that aid in 
interpolating missing color component information may be 15 
stored to a disk 318 or memory 311. It would be readily 
apparent to one of ordinary skill in the art of computer 
sciences to program a determining machine to carry out the 
interpolation methodology described in various embodi- 
ments of the invention. 20 

Computer system 310 has a system bus 313 which facili- 
tates information transfer to/from the processor 312 and 
memory 311 and also has a bridge 314 which couples system 
bus 313 to an I/O bus 315. I/O bus 315 connects various I/O 
devices such as a display adapter 316, disk 318 and an I/O 25 
port, such as a serial port. Many such combinations of I/O 
devices, buses and bridges can be utilized with the invention 
and the combination shown is merely illustrative of one such 
possible combination. 

When an image, such as an image of object 340, is 30 
captured, the image is sensed as R, G, and B pixels, in a CFA 
such as the Bayer pattern of FIG. 2. These pixel values are 
sent to the image processing circuit 332. Image processing 
circuit 332 consists of ICs and other components which 
execute different image processing functions. The need for 35 
color interpolation is most significant when the image is to 
be displayed or output and thus does not have to be per- 
formed on camera 330. When the user or application desires/ 
requests a download of images, the sub-sampled color 
images stored in the image memory unit are transferred from 40 
image memory unit 334 to the I/O Port 317. I/O port 317 
uses the bus-bridge hierarchy shown (I/O bus 315 to bridge 
314 to system bus 313) to temporarily store the image data 
into memory 311 or, optionally, to disk 318. 

The downloaded image will thus have pixels with missing 45 
color component information. If a full color image (each 
pixel having Red, Green and Blue) is desired to be output on 
monitor 320, then color interpolation should be employed. 
The downloaded sub-sampled image is analyzed pixel-by- 
pixel and by applying the techniques discussed above, each 50 
raw image pixel is transformed into a full color pixel. Color 
interpolated image 350 will appear to have a full 24-bit color 
resolution given adequate display devices (monitor 320 and 
adapter 316) will resemble the color quality of the original 
object 340. The interpolated missing color component val- 55 
ues and the raw image pixel values obtained from the Bayer 
pattern CFA may be stored onto disk 318, memory 311 or 
output directly to monitor 320 after being rendered through 
display adapter 316. 

The computer program code for color interpolation may 60 
be packaged onto a removable medium such as floppy disk 
or CD-ROM and may accompany software that performs the 
decompression of images downloaded from camera 330. As 
with any other software, it can be downloaded/distributed 
over a network on communications system, or can be 65 
available in firmware. These downloaded images can be 
color interpolated and stored into a useful full RGB format 
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so that the images will more accurately visually present the 
object 340 to the user. Color interpolation can also be 
implemented in hardware where no computer system 
capable of executing software code is available. 

In an alternative embodiment of the invention, the image 
processing circuit 332 can be adapted to include circuitry for 
performing color interpolation on-camera, such that the 
stored image in image memory unit 334 is a full color image. 
In such an embodiment, the image may be downloaded to a 
printer or other display device directly from camera 330 
without need of an intervening data processing system like 
computer system 310. The downloaded image from camera 
330 will already have full color since color interpolation, in 
this alternative embodiment of the invention, is performed 
on-camera during image processing. 

FIG. 4 illustrates a simplified methodology for determin- 
ing color median according to one embodiment of the 
invention. 

First, when a color median is to be determined, the 
relevant block of pixels for which the median is being 
determined should be defined (step 410). When these pixel 
values are obtained, they may be stored in a list or array. The 
median is defined as a value M in a set of values wherein one 
half of all members in the set lie below M and the other half 
lies above M. Thus, the median is a determination based on 
the distribution of values. As such, the pixel values in the 
defined block should be ordered from lowest to highest (step 
420). The ordered set {X^ X^, . . . , X^} is such that 
Xi^Xi+1 for all i ranging from 1 to N-l. 

Since the maximum number of pixels in any block 
according to the Bayer pattern and the embodiments dis- 
cussed above is 4, and the minimum is 3, the median 
determination can be simplified. If N-3, indicating that there 
are 3 pixels in the block, i.e., {X^ X^, X3} (checked at step 
430), then the median is (step 440). If N is not 3, then N 
must be 4 in at least one embodiment of the invention. If 
X2=X 3 in the ordered set {X u Xj, X 3 , X 4 }(checked at step 
450) then the median again is X^step 440). Otherwise, X 2 is 
different in value from X 3 , and the median becomes the 
average (midpoint) between the values X% and X 3 (step 460). 
The above methodology can be utilized where a Bayer 
pattern of 8 bits (one color per pixel in the original CFA) is 
used and where only adjacent pixels are candidates from 
which an interpolated value is selected or for which medians 
are determined. The methodology of FIG. 4 can be readily 
modified to adapt to a different image array pattern, as can 
the interpolation techniques in the other embodiments 
described earlier in this detailed description. 

The exemplary embodiments described herein are pro- 
vided merely to illustrate the principles of the invention and 
should not be construed as limiting the scope of the inven- 
tion. Rather, the principles of the invention may be applied 
to a wide range of systems to achieve the advantages 
described herein and to achieve other advantages or to 
satisfy other objectives as well. 

What is claimed is: 

1. A method of color interpolation for a raw image 
composed of pixels, comprising: 

interpolating missing color components for a raw image 
pixel by selecting neighboring pixels based on color 
medians for blocks of Green pixels surrounding said 
neighboring pixels; and 

forming a new full color pixel having full color 
information, said full color information a function of 
said raw image pixel and said interpolated color com- 
ponents; 

wherein said interpolating comprises: 
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determining if said raw image pixel is Green; 

if said raw image pixel is Green, applying a first 
interpolation methodology for interpolating Red and 
Blue missing color components, wherein the first 
interpolation methodology includes determining the 5 
color median of blocks surrounding adjacent Blue 
pixels and the color median of blocks surrounding 
adjaceDt Red pixels, and determining which adjacent 
Blue pixel has a surrounding block color median 
closest to the value of said raw image pixel and 10 
which adjacent Red pixel has a surrounding block 
color median closest to the value of said raw image 
pixel, and 

if said raw image pixel is not Green, applying a second 
interpolation methodology for interpolating a miss- 15 
ing Green color component and one of a missing 
Blue and Red color component. 

2. A method according to claim 1 wherein all said blocks 
are blocks of Green pixels. 

3. A method according to claim 1 wherein the color 20 
median of a block is determined by ordering pixel values 
within that block from lowest to highest. 

4. A method according to claim 1 further comprising: 
interpolating the missing Red component of said raw 

image pixel to be the Red pixel whose surrounding 25 
block color median is determined to be the closest to 
the value of said raw image pixel; and 
interpolating the missing Blue component of said raw 
image pixel to be the Blue pixel whose surrounding 
block color median is determined to be the closest to 30 
the value of said raw image pixel. 

5. A method according to claim 1 wherein said second 
interpolation methodology includes: 

interpolating the missing Green component as the median ^ 
of values of Green pixels adjacent to said raw image 
pixel. 

6. A method according to claim 1 further comprising: 
determining the color median for a block surrounding said 

raw image pixel; and 4Q 
determining the color median for a block surrounding 
each adjacent non-Green pixel surrounding said raw 
image pixel. 

7. A method according to claim 6 further comprising: 
determining which of said color medians for adjacent 45 

non-Green pixels is the closest to said color median for 
a block surrounding said raw image pixel; and 
interpolating the missing non-Green color component as 
the value of non- Green pixel whose color median is 
determined to be the closest to the color median of the so 
block surrounding said raw image pixel. 

8. A method according to claim 1 wherein raw image 
pixels located in image comers have non-Green color com- 
ponents interpolated to be adjacent non-Green pixel values. 

9. A method according to claim 1 wherein said image is 55 
arranged in a Bayer pattern. 

10. A method according to claim 1 wherein interpolating 
and forming are repeatedly performed for each pixel con- 
tained in said raw image. 

11. A system comprising: 60 
a data processing unit for coupling to an image capture 

device to provide raw image data, said data processing 
unit to interpolate missing color components for said 
raw image data by a color median-based selection 
technique, said technique including: 65 
interpolating missing color components for a raw 
image pixel by selecting neighboring pixels based on 



color medians for blocks of Green pixels surround- 
ing said neighboring pixels; and 
forming a new full color pixel having full color 
information, said full color information a function of 
said raw image pixel and said interpolated color 
components; 
wherein said interpolating comprises: 

determining if said raw image pixel is Green; 
if said raw image pixel is Green, applying a first 
interpolation methodology for interpolating Red 
and Blue missing color components, the first inter- 
polation methodology includes determining the 
color median of blocks surrounding adjacent Blue 
pixels and the color median of blocks surrounding 
adjacent Red pixels, determining which adjacent 
Blue pixel has a surrounding block color median 
closest to the value of said raw image pixel and 
which adjacent Red pixel has a surrounding block 
color median closest to the value of said raw 
image pixel; and 
if said raw image pixel is not Green, applying a 
second interpolation methodology for interpolat- 
ing a missing Green color component and one of 
a missing Blue and Red color component. 

12. A system according to claim 11 further comprising a 
storage medium coupled to said data processing unit, said 
storage medium storing said missing color components. 

13. A system according to claim 11 further comprising a 
display device coupled to said data processing unit, said 
display device configured to render raw image data together 
with interpolated color components to form a full color 
image upon output from said display device. 

14. Asystem according to claim 11 wherein all said blocks 
are blocks of Green pixels. 

15. A system according to claim 11 wherein the data 
processing unit is to interpolate using the color median of a 
block that is determined by ordering pixel values within that 
block from lowest to highest, 

16. A system according to claim 11 wherein the data 
processing unit is to interpolate, the interpolation further 
comprising: 

interpolating the missing Red component of said raw 
image pixel to be the Red pixel whose surrounding 
block color median is determined to be the closest to 
the value of said raw image pixel; and 

interpolating the missing Blue component of said raw 
image pixel to be the Blue pixel whose surrounding 
block color median is determined to be the closest to 
the value of said raw image pixel. 

17. A system according to claim 11 wherein the data 
processing unit is to interpolate, in said second interpolation, 
includes: 

interpolating the missing Green component as the median 
of values of Green pixels adjacent to said raw image 
pixel. 

18. A system according to claim 11 wherein the data 
processing unit is to interpolate, by further: 

determining the color median for a block surrounding said 

raw image pixel; and 
determining the color median for a block surrounding 

each adjacent non-Green pixel surrounding said raw 

image pixel. 

19. A system according to claim 18 wherein the data 
processing unit is to interpolate, by further: 

determining which of said color medians for adjacent 
non-Green pixels is the closest to said color median for 
a block surrounding said raw image pixel; and 
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interpolating the missing non-Green color component as 
the value of non-Green pixel whose color median is 
determined to be the closest to the color median of the 
block surrounding said raw image pixel. 

20. A system according to claim 11 wherein raw image 5 
pixels located in image comers have non-Green color com- 
ponents interpolated to be adjacent non-Green pixel values. 

21. A system according to claim 11 wherein said raw 
image data is arranged in a Bayer pattern. 

22. A system according io claim 11 wherein interpolating 30 
and forming are repeatedly performed for each pixel con- 
tained in said raw image. 

23. An article of manufacture comprising: 

a machine readable medium having instructions stored 
thereon which when executed by a set of processors 1S 
cause: 

missing color components for a raw image pixel to be 
interpolated by selecting neighboring pixels based 
on color medians for blocks of Green pixels sur- 
rounding said neighboring pixels; and 2 q 

a new full color pixel having full color information to 
be formed, said full color information a function of 
said raw image pixel and said interpolated color 
components; 

wherein said interpolating comprises: 25 
determining if said raw image pixel is Green; 
if said raw image pixel is Green, applying a first 
interpolation methodology for interpolating Red 
and Blue missing color components, the first inter- 
polation methodology to include determining the 30 
color median of blocks surrounding adjacent Blue 
pixels, the first interpolation methodology to fur- 
ther include determining which adjacent Blue 
pixel has a surrounding block color median closest 
to the value of said raw image pixel and deter- 35 
mining which adjacent Red pixel has a surround- 
ing block color median closest to the value of said 
raw image pixel; and 
if said raw image pixel is not Green, applying a 
second interpolation methodology for interpolat- 40 
ing a missing Green color component and one of 
a missing Blue and Red color component. 

24. An article of manufacture according to claim 23 
wherein all said blocks are blocks of Green pixels. 

25. An article of manufacture according to claim 23 45 
wherein the instructions further cause the color median of a 
block to be determined by ordering pixel values within that 
block from lowest to highest, 

26. An article of manufacture according to claim 23 
wherein the instructions further cause: 50 

the missing Red component of said raw image pixel to be 
interpolated to be the Red pixel whose surrounding 
block color median is determined to be the closest to 
the value of said raw image pixel; and 

the missing Blue component of said raw image pixel to be 55 
interpolated to be the Blue pixel whose surrounding 
block color median is determined to be the closest to 
the value of said raw image pixel. 

27. An article of manufacture according to claim 23 
wherein the instructions further cause said second interpo- eo 
lation to include: 

interpolating the missing Green component as the median 
of values of Green pixels adjacent to said raw image 
pixel. 

28. An article of manufacture according to claim 23 65 
wherein the instructions further cause the interpolation 
methodology to comprise: 



determining the color median for a block surrounding said 

raw image pixel; and 
determining the color median for a block surrounding 

each adjacent non-Green pixel surrounding said raw 

image pixel. 

29, An article of manufacture according to claim 28 
wherein the instructions further cause the interpolation 
methodology to comprise: 

determining which of said color medians for adjacent 
non-Green pixels is the closest to said color median for 
a block surrounding said raw image pixel; and 

interpolating the missing non-Green color component as 
the value of non-Green pixel whose color median is 
determined to be the closest to the color median of the 
block surrounding said raw image pixel. 

30, An article of manufacture according to claim 23 
wherein raw image pixels located in image corners have 
non- Green color components interpolated to be adjacent 
non- Green pixel values. 

31, An article of manufacture according to claim 23 
wherein said instructions are to color interpolate a raw 
image that is arranged in a Bayer pattern. 

32. An article of manufacture according to claim 23 
wherein instructions further cause interpolating and forming 
to be repeatedly performed for each pixel contained in a raw 
image. 

33, A method for color interpolation of an image com- 
posed of pixels, comprising: 

interpolating missing color components for a raw image 
pixel by selecting neighboring pixels based on Green 
pixels surrounding the neighboring pixels, wherein if 
said raw image pixel is Green, then applying a first 
interpolation methodology for interpolating Red or 
Blue missing color components, and if said raw image 
pixel is not Green, then applying a second interpolation 
methodology for interpolating a missing Green color 
component and a missing one of Blue or Red color 
components, 

the first interpolation methodology includes determining 
the color median of blocks of surrounding adjacent 
Blue pixels and blocks of surrounding adjacent Red 
pixels, 

and wherein said interpolating further includes determin- 
ing the color median for a block surrounding said raw 
image pixel and a block surrounding each adjacent 
non-Green pixel surrounding said raw image pixel, 

determining which of said color medians for adjacent 
non-Green pixels is the closest to said color median for 
a block surrounding said raw image pixel, and inter- 
polating a missing non-Green color component as the 
value of the non-Green pixel whose color median is 
determined to be closest to the color median of the 
block surrounding said raw image pixel. 

34. The method of claim 33 wherein the first interpolation 
methodology further includes determining which adjacent 
Blue pixel has a surrounding block color median closest to 
the value of said raw image pixel, and determining which 
adjacent Red pixel has a surrounding block color median 
closest to the value of said raw image pixel. 

35. The method of claim 33 wherein the color median of 
a block is determined by ordering pixel values within that 
block from lowest to highest. 

36. The method of claim 33 further comprising: 
interpolating the missing Red component of said raw 

image pixel to be the Red pixel whose surrounding 
block color median is determined to be the closest to 
the value of said raw image pixel; and 
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interpolating the missing Blue component of said raw 
image pixel to be the Blue pixel whose surrounding 
block color median is determined to be the closest to 
the value of said raw image pixel. 

37. The method of claim 33 wherein said second inter- 5 
polation methodology includes interpolating the missing 
Green component as the median of values of Green pixels 
adjacent to said raw image pixel. 

38. The method of claim 33 wherein raw image pixels 
located in image comers have non-Green color components 10 
interpolated to be adjacent non-Green pixel values. 

39. The method of claim 33 wherein said image is 
arranged in a Bayer pattern. 

40. The method of claim 33 wherein interpolating and 
forming are repeatedly performed for each pixel contained 15 
in said image. 

41. An article of manufacture comprising: 

a machine readable medium having instructions stored 
thereon which when executed by one or more proces- 
sors cause 20 
missing color components for a raw image pixel to be 
interpolated by selecting neighboring pixels based 
on color medians for blocks of Green pixels sur- 
rounding said neighboring pixels; and 
a new full color pixel having full color information to 25 
be formed, said full color information being a func- 
tion of said raw image pixel and said interpolated 
color components, wherein if said raw image pixel is 
Green, said interpolation includes applying a first 
interpolation methodology for interpolating Red and 30 
Blue missing color components and if said raw 
image pixel is not Green, applying a second inter- 
polation methodology for interpolating a missing 
Green color component and a missing one of Blue 
and Red color components, 35 
the first interpolation methodology to further include 
determining the color median of blocks surrounding 
adjacent Blue pixels and the color median of blocks 
surrounding adjacent Red pixels, 
and wherein said interpolation further includes deter- 40 
mining the color medians for a block surrounding 
said raw image pixel and a block surrounding each 
adjacent non- Green pixel surrounding said raw 



image pixel, the interpolation methodology to further 
include determining which of said color medians for 
adjacent non-Green pixels is the closest to said color 
median for a block surrounding said raw image pixel 
and interpolating a missing non-Green color com- 
ponent as the value of a non-Green pixel whose color 
median is determined to be the closest to the color 
median of the block surrounding said raw image 
pixel. 

42. The article of manufacture of claim 41 wherein the 
instructions further cause the first interpolation methodology 
to further include: 

determining which adjacent Blue pixel has a surrounding 

block color median closest to the value of said raw 

image pixel and 
determining which adjacent Red pixel has a surrounding 

block color median closest to the value of said raw 

image pixel. 

43. The article of manufacture of claim 41 wherein the 
instructions further cause: 

the missing Red component of said raw image pixel to be 
interpolated to be the Red pixel whose surrounding 
block color median is determined to be the closest to 
the value of said raw image pixel; and 

the missing Blue component of said raw image pixel to be 
interpolated to be the Blue pixel whose surrounding 
block color median is determined to be the closest to 
the value of said raw image pixel, 

44. The article of manufacture of claim 41 wherein the 
instructions further cause said second interpolation to 
include: 

interpolating the missing Green component as the median 
of values of Green pixels adjacent to said raw image 
pixel. 

45. The article of manufacture of claim 41 wherein said 
instructions are to color interpolate a raw image that is 
arranged in a Bayer pattern. 

46. The article of manufacture of claim 41 wherein said 
instructions further cause interpolating and forming to be 
repeatedly performed for each pixel contained in a raw 
image. 
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UNITED STATES PATENT AND TRADEMARK OFFICE 

CERTIFICATE OF CORRECTION 



PATENT NO. : 6,269,181 Bl Page 1 of 1 

DATED : July 31, 2001 

INVENTOR(S) : Acharya 



It is certified that error appears in the above-identified patent and that said Letters Patent is 
hereby corrected as shown below: 



Column 7, 

Line 54, delete "(Green)", insert - P(Green) ». 



Signed and Sealed this 



Fourth Day of June, 2002 



Attest: 




JAMES E. ROGAN 

Attesting Officer Director of the United States Patent and Trademark Office 
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